Ergodic and adaptive control of hidden Markov models

نویسندگان

Tyrone E. Duncan

Bozenna Pasik-Duncan

Lukasz Stettner

چکیده

A partially observed stochastic system is described by a discrete time pair of Markov processes. The observed state process has a transition probability that is controlled and depends on a hidden Markov process that also can be controlled. The hidden Markov process is completely observed in a closed set and only observed through the other process in the complement of the closed set. Initially an ergodic control problem is formulated and solved by a vanishing discount approach. An equation is given for the optimal cost and an optimal control for the ergodic control problem that is obtained from the family of Bellman equations for the discounted problems. For the ergodic control problem, the closed set where the hidden Markov process is completely observed can be the empty set. For an adaptive control problem, it is assumed that the transition operators for the observed state process and the hidden Markov process depend on a parameter from a compact metric space. It is shown that the optimal cost is a continuous function of the parameter and there is a control that is almost optimal for a parameter in an open set. It is assumed that the unknown parameter is a random variable. A sequence of estimates of this parameter are given which is modified at random times by the use of a control from a finite family. It is ∗Research supported in part by NSF Grants DMS 0204669 and ANI 0125410. †Department of Mathematics, University of Kansas, Lawrence, KS 66045, duncan@ math.ukans.edu and [email protected] ‡Institute of Mathematics, Polish Academy of Sciences, Warsaw, Poland, stettner@ impan.gov.pl.

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

An Adaptive Approach to Increase Accuracy of Forward Algorithm for Solving Evaluation Problems on Unstable Statistical Data Set

Nowadays, Hidden Markov models are extensively utilized for modeling stochastic processes. These models help researchers establish and implement the desired theoretical foundations using Markov algorithms such as Forward one. however, Using Stability hypothesis and the mean statistic for determining the values of Markov functions on unstable statistical data set has led to a significant reducti...

متن کامل

Introducing Busy Customer Portfolio Using Hidden Markov Model

Due to the effective role of Markov models in customer relationship management (CRM), there is a lack of comprehensive literature review which contains all related literatures. In this paper the focus is on academic databases to find all the articles that had been published in 2011 and earlier. One hundred articles were identified and reviewed to find direct relevance for applying Markov models...

متن کامل

A Non-Equilibrium Analysis & Control Framework for Active Queue Management

We present a non-equilibrium analysis and control approach for the Active Queue Management (AQM) problem in communication networks. Using simplified fluid models, we carry out a bifurcation study of the complex dynamic queue behavior to show that non-equilibrium methods are essential for analysis and optimization in the AQM problem. We investigate an ergodic theoretic framework for stochastic m...

متن کامل

Ergodicity of hidden Markov models

In this paper we study ergodic properties of hidden Markov models with a generalized observation structure. In particular sufficient conditions for the existence of a unique invariant measure for the pair filter-observation are given. Furthermore, necessary and sufficient conditions for the existence of a unique invariant measure of the triple state-observation-filter are provided in terms of a...

متن کامل